Skip to content

Add SCIM identity providers with group-to-role mapping - #1464

Merged
jackmusick merged 29 commits into
gobifrost:mainfrom
MTG-Thomas:feat/scim-identity-providers
Nov 18, 2026
Merged

jackmusick merged 29 commits into
gobifrost:mainfrom
MTG-Thomas:feat/scim-identity-providers

Conversation

@MTG-Thomas

@MTG-Thomas MTG-Thomas commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Product Updates

  • Highlight. This is a new user-facing feature. Approved entry: 182c8b4f-4cb8-467c-bc96-0c24b16106b7.
  • Other
  • Omit

Security review requested: this adds a public, bearer-authenticated endpoint that writes
users and role assignments.

Summary

Each organization, the provider org included, can connect one or more identity
providers. A connection keeps Bifrost users and their roles in step with an external
directory over SCIM 2.0. The UI follows NinjaOne's identity provider screens. Sign-in is
unchanged and stays instance-wide.

The v1 scope is organization-bound connections limited to that organization's verified domains. Maintainer approval of the exact Product Update and protected delivery is recorded in the native owner conversation.

What it adds

  • Connections: Users → Identity Providers lists connections. Each one has its own
    hashed bearer token with an expiry. Rotating the token keeps the old one valid for one
    more hour.
  • Domain claims: an organization proves it owns an email domain with a DNS TXT
    record. Only one organization can verify a given domain.
  • SCIM 2.0 endpoint: /api/scim/v2/{identity_provider_id}.
    • Discovery, Users and Groups.
    • The eq filters that identity providers actually send.
    • Paging.
    • Entra's and Okta's PATCH dialects.
    • The full payload, extensions included, is stored on the user's link.
    • Validation and patching use scim2-models.
  • Provisioning core: independent of the SCIM protocol, so a Google Directory sync can
    reuse it later.
    • Adoption: an existing user in the same org is adopted by email. A user in another org
      gets 409 uniqueness.
    • Deactivation: active=false deactivates the user and revokes their sessions.
    • DELETE deactivates and unlinks the user. It hard-deletes only a user the connection
      created who has never signed in, which Google's connection test needs.
  • Group-to-role mapping:
    • Revocation: a connection removes only the roles it granted.
    • Mapped groups: once a connection has any mapping, a user must be in at least one
      mapped group to stay active.
    • Escalation check: creating a mapping requires that the caller could assign those
      roles by hand.
  • Grants wait for the access pass: revocations apply immediately. Grants and
    activations are applied by a scheduled access pass every two minutes, which caps how
    often the global authz epoch is bumped. Users waiting on the pass appear as
    "Activating" in the Users list.
  • Credential expiry: an hourly check keeps a platform-admin notification up while a
    token is within 14 days of expiry, or past it. The notification has a Regenerate Token
    action. It also emits identity_provider.token_expiry, so a workflow can tell the
    organization's own admins.
  • Events: user.provisioned, user.updated, user.deprovisioned,
    user.groups_changed and user.access_changed, alongside the existing user.created
    and user.signed_in.
  • Permissions: a new identityproviders permission domain (read and readwrite),
    checked at the connection's organization.
  • Catalogs: the operation catalog, the access list and the generated client types are
    updated.

The design doc is docs/plans/2026-11-16-scim-identity-providers.md. It covers the
decisions, the lock order, notes for each identity provider, and open questions.

Rules that limit a directory's authority (decisions 13 and 14)

  • Privileged access stays a Bifrost decision:
    • A directory never adopts a privileged user (one in privileged_user_ids, Platform
      Admin included).
    • A role carrying privileged permissions can't be mapped.
    • Each grant is checked again when it applies, in case a role became privileged after
      it was mapped.
    • A linked user who later becomes privileged is revoke-only. The directory can
      still deactivate or unlink them and remove the roles it granted, so offboarding
      always works. It can't re-activate, rename or re-email them, or grant them anything.
    • Note: PRIVILEGED_PERMISSIONS is broad. It includes tables.readwrite,
      executions.readwrite and code.execute, so roles carrying any of these have to be
      assigned in Bifrost.
  • No moves out of the org: a user an identity provider manages can't be moved to
    another organization. That covers a single move, a bulk move, and a promotion to
    Platform Admin; each returns 409.

Changes to shared code (please look at these)

  • shared/account_guards.py:
    • assert_role_assignment_allowed_for_user and assert_user_can_be_external now take
      FOR SHARE role locks (share_lock_roles) instead of FOR UPDATE.
    • The permission editor, the last-admin guard and the SSO policy keep FOR UPDATE.
    • Read-only checks no longer serialize against each other. A permission edit still
      excludes them.
  • shared/sdk_users.py:
    • update_user and delete_user lock the user row before any guard locks a role.
    • bulk_update_users loads users FOR UPDATE in id order.
  • shared/sdk_roles.py and services/user_role_assignments.py: role-assignment writes
    also lock the user first.
  • core/transaction_conflicts.py: conflict_aware_route gains an on_duplicate mapping
    for unique violations. It logs only the SQLSTATE and the constraint name.
  • services/notification_service.py: adds find_admin_notification_by_metadata.

Sensitive paths

  • Auth: the new public SCIM endpoint and its bearer tokens.
  • Multi-tenancy: domain claims and the org-boundary checks.
  • Migrations: three new revisions.
  • Audit: every provisioning write is audited, with the connection as the actor.
  • New dependencies: scim2-models, lark, dnspython (now direct), and scim2-tester
    (dev only).

Size and splitting: about 13.4k added lines.

Part Added lines
API source 5.4k
Client source 2.3k
Tests 3.3k
Generated files and lockfiles 2.1k
Design doc 0.4k

I'm happy to split this into a stack if that's easier to review:

  1. domain claims;
  2. the provisioning core and SCIM endpoint;
  3. token expiry and the access pass;
  4. the UI.

Validation (suites on a55129bd2, merged with current main; 830cff527 adds
only the schema fix, with its unit test and ./test.sh quality api rerun)

  • ./test.sh unit (full backend unit suite): 13,578 passed, 0 failed.
  • ./test.sh quality api (pyright and ruff): clean.
  • npm run tsc: clean. npm run lint: no new errors.
  • ./test.sh client unit for the identity-provider pages, badges and service:
    passed.
  • ./test.sh tests/e2e/api/test_scim_provisioning.py, which includes the
    scim2-tester discovery and Group CRUD checks: 2 passed.
  • ./test.sh client e2e e2e/identity-providers.admin.spec.ts: 2 passed.
  • Two-session database concurrency tests (test_identity_provider_concurrency.py) cover:
    • a move racing adoption;
    • a demotion racing a deactivation;
    • an admin and a directory deactivating at once;
    • a grant racing a permission edit;
    • token rotation;
    • a mapping racing a permission edit.
  • Live check: a dev stack exposed only /api/scim/v2/ through a temporary tunnel.
    • An Entra-shaped lifecycle passed over the public URL: create (201), duplicate
      (409), filter, "False" PATCH, delete (204).
    • Microsoft's SCIM Validator fetched /Schemas and then refused to run, because we
      advertised password. Bifrost drops passwords, so discovery no longer lists it
      (830cff5).
    • A full validator run hasn't been completed. Its UI kept the old password row,
      and the validator's network access was unreliable. No real Entra or Okta tenant
      has been connected yet.
  • Review: four adversarial review rounds by two independent reviewers. Every high- and
    medium-severity finding is fixed.

Known limitations, accepted

  • PostgreSQL can detect deadlocks for External users in large access-pass batches. The
    affected link is retried on the next pass.
  • A PATCH that targets an extension Bifrost hasn't stored yet can be filed under a
    slightly wrong schema URN.
  • An expiry event can arrive just after the token was rotated.
  • The email lookup uses lower(email) without an index, the same as the existing SSO
    sign-in lookup.

Not in this PR: Google Directory sync; group claims at sign-in; per-org sign-in; SAML;
trusting proxy headers.

Contribution Basis And Tracking

Fixes #1463.

Basis: maintainer-requested. On 2026-11-16, @jackmusick said he'd "definitely like"
SCIM and group sync from an identity provider, modeled on NinjaOne's per-org
connections. He then said: "if you want to open a issue for this against that thread I'm
more than happy to give my blessing on how Ninja does it... This is clearly the way."

Related: #1361 (the sign-in side, still open).

🤖 Generated with Claude Code

Focused delivery corrections

  • Directory deletion of a mapped group returns 409 until an administrator explicitly unmaps it, preserving the mapped-group access requirement.
  • Applied directory role grants and revocations write transactional audit records with connection attribution and affected role IDs, including when the user remains active.
  • Product Update evidence cites Jack's native exact-text approval; the approved title/body and contributor attribution are preserved.

Focused local verification: 51 selected core, access-pass, concurrency, authentication, organization lifecycle and live SCIM tests; API pyright and ruff. Full suites remain the protected merge-queue gate.

MTG-Thomas and others added 23 commits November 16, 2026 12:29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Organizations can connect an identity provider (Entra ID, Okta, JumpCloud,
Google Workspace's SCIM beta) that provisions their users over SCIM 2.0 at
/api/scim/v2/{identity_provider_id}:

- Domain claims verified by DNS TXT gate which users a connection may
  create or change; a domain is verified by one organization only.
- Existing same-org users are adopted by email; users of another org are a
  uniqueness conflict.
- Directory groups map to roles. Provisioning tags the assignments it makes
  (user_roles.identity_provider_id) and only removes those, and once a
  connection has a mapping a user needs a mapped group to stay active.
- DELETE disables and unlinks; a user the connection created who never
  signed in is removed outright.
- Tokens are stored hashed and rotate with a one-hour overlap.

The provisioning core is protocol-independent so a Google Directory sync can
feed it later. scim2-models validates resources, applies PatchOp and parses
filters; requests are normalized for Entra and Okta quirks first.
Administration is a new identityproviders permission domain, evaluator-gated
and catalogued; provisioning emits user.provisioned / updated /
deprovisioned / groups_changed events and audit rows naming the connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rename the group-mapping role model to MappedRole: as RoleSummary it
collided with the role-assignment schema of the same name, and OpenAPI
silently kept only one of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record how several connections in one organization behave and what v1
deliberately leaves open for enterprise IT. Following the community thread,
add the Access-for-SaaS sign-in check, feed sign-in group claims into the
same mappings, and lead the upstream RFC with per-organization sign-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Users gains an Identity Providers view, modeled on NinjaOne's:

- The list shows each connection's organization, provider, verified
  domains, provisioning status, token expiry, users and groups mapped.
- The detail page has the SCIM card (Tenant URL to copy, token hint and
  expiry, regenerate with a one-time reveal), the organization's email
  domains with the TXT record to publish and a Verify action, and the group
  mapping table (group, mapping status, roles; a mapping with no roles
  grants access only).
- An expired or expiring token, or no verified domain, raises a banner.
- Editing uses a dialog with a left-hand section nav (General,
  Provisioning).
- The Users list gains an Identity Source column: the managing connection,
  or Native.

Covered by service and page vitest suites and a Playwright happy path that
creates a connection, pushes a group over SCIM with the revealed token, and
maps it to a role.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An expired SCIM token silently stops provisioning. An hourly scheduled task
now checks each enabled connection's current token (superseded tokens in
their rotation overlap are ignored):

- Within 14 days of expiry, or past it, it keeps a platform-admin
  notification up with a Regenerate Token action. Notifications live an hour
  in Redis, so each run re-raises one that lapsed.
- It emits identity_provider.token_expiry once per stage (expiring, then
  expired), recorded on the token, so a workflow can reach the
  organization's own admins.
- Rotating the token dismisses the outstanding warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deleting an organization only disables it, so its verified domain claims
stayed and kept every other organization from verifying those domains.
Disabling now releases the claims (reactivating means verifying again,
since control of a domain may have changed), and a disabled organization's
connections stop: SCIM requests are refused and the expiry check skips
them, resuming if the organization is reactivated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bifrost has one global authority epoch: any committed change to a user's
activity or name, or to a role assignment, retires every access token on
the instance. Applied request by request, a 2,000-user SCIM initial sync
did that about 2,000 times in 40 minutes.

Provisioning now revokes in the request and defers grants. Activation,
new roles and renames are recorded on the link (access_pending_since,
display_name) and applied by the access pass, a scheduled task every two
minutes that applies up to 500 waiting links, oldest first, in one
transaction: at most one epoch change per run. New users are inserted in
their final state, since an insert doesn't change the epoch. Admin
mapping changes still apply at once.

Concurrency, from two GPT-6.1 Sol reviews:
- A per-connection advisory policy lock: shared for provisioning writes,
  exclusive for mapping changes and group deletion, always before user
  locks. The pass takes every involved connection's lock up front.
- Users are locked in id order before links, memberships and assignments
  are read; SCIM PUT/PATCH/DELETE lock before reading what they change.
- Deleting the last mapped group reconciles every linked user.
- Cache invalidation runs after commit, once per role.
- A failing link rolls back in its own savepoint and is re-queued; a
  policy-refused grant is audited.

user.groups_changed now reports membership only; the new
user.access_changed reports applied activation and role changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
set_group_members deleted membership rows before locking the members'
users, while delete_user locks the user first and then cascades into
those rows: two concurrent requests could deadlock. Lock the affected
users first, in id order, then re-read the member list. Found by
GPT-6.1 Sol's verification review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL breaks the rare remaining deadlocks (the access pass and a
mapping change on another connection granting one role to External
users, or two mapping changes) by rolling one transaction back. That
surfaced as a generic 500 or an exception log.

Now each surface classifies the abort by SQLSTATE (40P01, 40001, via a
new is_transaction_conflict shared with the database retry helper),
logs a warning and answers explicitly:
- SCIM: 503 with Retry-After: 5, which identity providers retry.
- Management API: 409 "nothing was saved; try again".
- Access pass: a link rolled back counts as a conflict and waits for the
  next run; a whole-run abort is logged and left to the next run.

A test drives a real deadlock between two sessions to prove the
classifier matches the error asyncpg raises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grants from an identity provider now wait up to two minutes for the
access pass, so a user Entra just added can look inactive with no
explanation. The user's identity source carries access_pending_since
(Users list: an "Access Pending" badge with the reason), and a
connection reports pending_access_count (provisioning card: "N awaiting
access").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Users, Identities and Identity Providers tabs switch to cards below
1280px so their tables never scroll sideways. Identity Providers shows
the provider under the connection name, Users shows the identity
source under the user's name, and truncated emails no longer force the
email column to its full width.

A new hire an identity provider is activating is listed by default with
an Activating status until the access pass turns them on, instead of
hiding as an inactive user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-parents the identity provider migrations onto main's
20261117_integ_secret_encrypt so there is one Alembic head, and
regenerates both locks from the merged pyproject (main's flags and
Python 3.15; only the SCIM packages differ from main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge of origin/main committed conflict markers in
api/src/services/events/registry.py. Keep both sides: main's
user.created and user.signed_in topics and this branch's identity
provisioning topics, so all stay reserved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Privileged access is decided in Bifrost, never by a directory: a
connection won't adopt, change or delete a privileged user, the access
pass leaves one alone, and a role with privileged permissions can't be
mapped and is refused at grant time if it gains them later. An
IdP-managed user can't be moved to another organization (directly, in
bulk or by promotion), so a connection can't act or grant roles outside
its own organization.

Also:
- Emails and SCIM userNames match without regard to case, with a
  case-insensitive unique index on userName.
- A PUT that omits active keeps the user's current state.
- SCIM authenticates before rate limiting; bad tokens are limited per
  client and can't use up the connection's quota.
- Token rotation serializes on the connection, so two concurrent
  rotations can't both keep a full-lifetime token.
- Expiry warnings are matched by connection id, not name, and dismissed
  before the rotation commits.
- A competing domain verification answers 409 instead of a 500.
- PATCH rejects unsupported verbs before touching unknown extensions,
  and malformed input (no userName, non-object extensions, deep JSON,
  huge startIndex, a lost unique race) answers 400/409 instead of 500.
- Mapping changes read the group under the exclusive policy lock.
- Access Pending isn't shown for a disabled connection or organization.
- A newly issued token leaves browser history as soon as it's shown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Moves lock the user row before checking for a link, as adoption does,
  and check the move's real destination (an explicit organization wins
  over promotion). The provisioner also leaves alone a linked user who is
  no longer in its organization.
- Privilege decisions read role permissions under a share lock that a
  permission edit (which locks the role for update) waits on, both for
  the user being changed and for each role being granted.
- Adoption refuses to choose between accounts whose emails differ only
  in letter case.
- Conflict logs carry the SQLSTATE and constraint, not the statement,
  its parameters or the key values.
- Failed SCIM authentication is limited per connection rather than by
  X-Forwarded-For, which the client controls.
- Rotation reads the token it replaces under the issuing lock, so a
  concurrent rotation's token is the one reported.
- The expiry check raises its warnings under a key-share lock on the
  connections before committing, so a rotation can't land between the
  check and the warning; events still follow the commit.
- user.groups_changed reports the user as the change leaves them.
- Deeply nested bodies and non-object extension values answer 400, and
  a guard refusal during a mapping change answers 409.

Adds two-session tests for adoption against a move, a grant against a
permission edit, and two rotations at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision: a directory can still take access away from a linked
user who became privileged (deactivate, unlink, remove roles it
granted), since Bifrost's privileged permission set is broad and
offboarding must keep working. It still never adopts a privileged user,
re-activates, renames or re-emails one, or grants a privileged role.

From the adversarial review:
- The users API locks the user row before any guard locks a role
  (single update, delete, and every bulk operation), the order identity
  provisioning uses, so an admin and a directory deactivating the same
  user can't deadlock.
- The move test now holds adoption's lock with the link uncommitted, so
  it fails if the move doesn't wait and look; it covers bulk moves too.
- A role deleted during a grant is skipped instead of failing on its
  foreign key.
- Bad-token limits key on the parsed connection id, so other spellings
  of the same UUID share one quota.
- Adoption that omits active keeps the user's current state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the third review round:
- The assignment and External-transition guards only read role
  permissions, so they share-lock the role (shared.account_guards.
  share_lock_roles) instead of locking it for update. A permission edit
  still locks for update and waits for them; two readers no longer
  deadlock upgrading a lock both share, which External-user grants hit.
  Provisioning uses the same helper.
- Updating or deleting a user rereads it under its row lock before
  authorizing, and bulk operations load their users locked, so a
  concurrent move can't be authorized against the old organization.
- Mapping refuses a privileged role under a share lock held to the
  commit, so a role can't turn privileged between the check and the save.
- The expiry check takes FOR KEY SHARE (what key_share alone didn't
  emit) and reads tokens only for the connections it locked.
- POST and PUT refuse a non-object value for an unknown extension.

Adds two-session tests: two grants of one role to External users don't
wait for each other, and a mapping that races a permission edit is
refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the adversarial confirmation pass:
- A SCIM DELETE of a privileged user unlinks and deactivates them
  through the account guards; only an unprivileged, never-signed-in
  account the connection created is removed outright. Before, the hard
  delete skipped the last-admin and SSO-recovery checks.
- Provisioning no longer share-locks built-in roles (their permissions
  can't change). The last-admin guard locks Platform Admin for update
  during a deactivation, and a share lock taken first would have to be
  upgraded, deadlocking two concurrent admin offboardings.
- Extension attributes sent flattened at the top level, as Entra does
  for custom attributes ("<schema URN>:<attribute>": value), are filed
  under their schema (merged with the nested form for enterprise)
  instead of refused.

Adds a two-session test that an admin and a directory deactivating the
same user don't deadlock (single and bulk), which fails with the old
role-before-user order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Sol's confirmation pass:
- Role assignment writes (PUT, base-role change, SDK assign/unassign)
  lock the user before the last-admin guard locks Platform Admin, the
  order identity provisioning uses. Adds a two-session test that a
  Platform Admin demotion and a directory deactivation don't deadlock.
- An update or delete whose user was deleted while it waited for the
  lock answers 404 instead of failing on a stale object.

Found by running the full unit suite:
- Provisioning no longer reads is_superuser directly (privileged_user_ids
  already counts Platform Admin) or passes it to the User constructor;
  update_user's extra read is registered in the elevated-check inventory.
- The new notification lookup is registered as a plain Redis read.
- The import-boundary test lists identity_providers among the users
  contract's dependencies (UserPublic.identity_source).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bifrost drops a password on arrival and reports changePassword as
unsupported, so /Schemas no longer lists it. Microsoft's SCIM Validator
refused to run against the endpoint while it did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jackmusick
jackmusick enabled auto-merge November 18, 2026 21:16
@jackmusick
jackmusick added this pull request to the merge queue Nov 18, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026
@jackmusick
jackmusick enabled auto-merge November 18, 2026 22:03
@jackmusick
jackmusick added this pull request to the merge queue Nov 18, 2026
Merged via the queue into gobifrost:main with commit e22df7d Nov 18, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature]: SCIM provisioning and group-to-role mapping per organization

3 participants